Gesture recognition method and system using siamese neural network

ABSTRACT

A gesture recognition system using siamese neural network executes a gesture recognition method. The gesture recognition method includes steps of: receiving a first training signal to calculate a first feature; receiving a second training signal to calculate a second feature; determining a distance between the first feature and the second feature in a feature space; adjusting the distance between the first feature and the second feature in feature space according to a predetermined parameter. Two neural networks are used to generate the first feature and the second feature, and determine the distance between the first feature and the second feature in the feature space for training the neural networks. Therefore, the gesture recognition system does not need a big amount of data to train one neural network for classifying a sensing signal. A user may easily define a new personalized gesture.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to a recognition method and a recognitionsystem, and more particularly to a gesture recognition method and agesture recognition system using siamese neural network.

2. Description of the Related Art

Recognition systems generally receive sensing signals from a sensor torecognize the motion of the user by using a neural network. For example,the recognition system receives sensing signals from the sensor,processes the sensing signals using the neural network, and utilizes therecognition system to determine whether a user being observed by thesensor is using portions of his or her body to do particular actions orform particular shapes, or gestures. Then, the recognition systemclassifies the motion of the user, and associates the motion of the userwith executable commands or instructions.

However, the neural network needs a big amount of training data toensure accuracy of a recognition result of the recognition system. Whena user wants to define a personalized gesture, the user needs to do aparticular action for many times to train the neural network of therecognition system. It is very inconvenient for the user to define thepersonalized gesture by him.

Therefore, the recognition system needs to be further improved.

SUMMARY OF THE INVENTION

An objective of the present invention is to provide a gesturerecognition method and a gesture recognition system using siamese neuralnetwork. The present invention may train a neural network of arecognition system by a few training data. Therefore, the user mayeasily define a personalized gesture.

The gesture recognition method using siamese neural network comprisessteps of:

controlling weight of a first neural network unit and weight of a secondneural network unit to be the same by a weight sharing unit;

receiving a first training signal from a sensor to calculate a firstfeature by the first neural network unit;

receiving a second training signal from the sensor to calculate a secondfeature by the second neural network unit;

determining a distance between the first feature and the second featurein a feature space by a similarity analysis unit;

controlling the weight of the first neural network unit and the weightof the second neural network unit through the weight sharing unit toadjust the distance between the first feature and the second feature ina feature space according to a predetermined parameter by a weightcontrolling unit.

The gesture recognition system using siamese neural network comprises asensor, a first neural network unit, a second neural network unit, aweight sharing unit, a similarity analysis unit, and a weightcontrolling unit.

The sensor senses a first training signal and a second training signal.

The first neural network unit is electrically connected to the sensor toreceive the first training signal, and calculates a first feature. Thesecond neural network unit is electrically connected to the sensor toreceive the second training signal, and calculates a second feature.

The weight sharing unit is electrically connected to the first neuralnetwork unit and the second neural network unit, and controls weight ofthe first neural network unit and weight of the second neural networkunit to be the same.

The similarity analysis unit is electrically connected to the firstneural network unit and the second neural network unit, and receives thefirst feature and the second feature. The similarity analysis unitdetermines a distance between the first feature and the second featurein a feature space.

The weight controlling unit is electrically connected to the similarityanalysis unit and the weight sharing unit. The weight controlling unitcontrols the weight of the first neural network unit and the weight ofthe second neural network unit through the weight sharing unit to adjustthe distance between the first feature and the second feature in thefeature space according to a predetermined parameter.

The present invention can use two neural networks to generate twofeatures, and can determine the distance between the first feature andthe second feature in the feature space for training the first neuralnetwork unit and the second neural network unit. Therefore, the presentinvention does not need a big amount of data to train one neural networkfor classifying a sensing signal. The user may easily define a newpersonalized gesture.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flowchart of a first embodiment of a gesture recognitionmethod using siamese neural network of the present invention;

FIG. 2 is a block diagram of a first embodiment of a gesture recognitionsystem using siamese neural network of the present invention;

FIG. 3 is a flowchart of a second embodiment of a gesture recognitionmethod using the siamese neural network of the present invention;

FIG. 4 is a flowchart of a third embodiment of the gesture recognitionmethod using the siamese neural network of the present invention; and

FIG. 5 is a block diagram of a second embodiment of the gesturerecognition system using the siamese neural network of the presentinvention.

DETAILED DESCRIPTION OF THE INVENTION

With reference to FIG. 1, the present invention relates to a gesturerecognition method and a gesture recognition system using siamese neuralnetwork. The gesture recognition method includes steps of:

controlling weight of a first neural network unit and weight of a secondneural network unit to be the same by a weight sharing unit (S101);

receiving a first training signal from a sensor to calculate a firstfeature by the first neural network unit (S102);

receiving a second training signal from the sensor to calculate a secondfeature by the second neural network unit (S103);

determining a distance between the first feature and the second featurein a feature space by a similarity analysis unit (S104);

controlling the weight of the first neural network unit and the weightof the second neural network unit through the weight sharing unit toadjust the distance between the first feature and the second feature inthe feature space according to a predetermined parameter by a weightcontrolling unit (S105).

With reference to FIG. 2, the gesture recognition system using siameseneural network includes a sensor 10, a first neural network unit 11, asecond neural network unit 12, a weight sharing unit 13, a similarityanalysis unit 14, and a weight controlling unit 15.

The sensor 10 senses the first training signal and the second trainingsignal. The first neural network unit 11 is electrically connected tothe sensor 10 to receive the first training signal, and the first neuralnetwork unit 11 calculates the first feature according to the firsttraining signal.

The second neural network unit 12 is electrically connected to thesensor 10 to receive the second training signal, and the second neuralnetwork unit 12 calculates the second feature according to the secondtraining signal.

The weight sharing unit 13 is electrically connected to the first neuralnetwork unit 11 and the second neural network unit 12, and the weightsharing unit 13 controls the weight of the first neural network unit 11and the weight of the second neural network unit 12 to be the same.

The similarity analysis unit 14 is electrically connected to the firstneural network unit 11 and the second neural network unit 12, and thesimilarity analysis unit 14 receives the first feature and the secondfeature. The similarity analysis unit 14 determines a distance betweenthe first feature and the second feature in the feature space.

The weight controlling unit 15 is electrically connected to thesimilarity analysis unit 14 and the weight sharing unit 13. The weightcontrolling unit 15 controls the weight of the first neural network unitand the weight of the second neural network unit through the weightsharing unit 13 to adjust the distance between the first feature and thesecond feature in the feature space according to a predeterminedparameter.

The present invention can use two neural networks to generate twofeatures, and can determine a similarity between the two features fortraining the first neural network unit and the second neural networkunit. Therefore, the present invention does not need a big amount ofdata to train one neural network for classifying a sensing signal, andthe user may easily define a new personalized gesture.

For example, when the user wants to define a new personalized gesture,the user may perform the personalized gesture at least one time. Thesensor 10 may sense the personalized gesture to generate the firsttraining signal. The first neural network unit 11 may generate the firstfeature according to the first training signal. Then, the first featuremay be stored in a database as a reference anchor sample for furtherclassification.

Moreover, the user may perform another personalized gesture at least onetime. The sensor 10 may sense said another personalized gesture togenerate the second training signal. The second neural network unit 11may generate the second feature according to the second training signal.Then, the similarity analysis unit 14 may load the first feature storedin the database and receive the second feature to calculate the distancebetween the first feature and the second feature in the feature space.The similarity analysis unit 14 may load the first feature stored in thedatabase and receive the second feature to calculate the distancebetween the first feature and the second feature in the feature space.Then, the weight controlling unit 15 controls the weight of the firstneural network unit 11 and the weight of the second neural network unit12 for training the first neural network unit 11 and the second neuralnetwork unit 12 according to the predetermined parameter.

Further, with reference to FIG. 3, in the step S105 of the gesturerecognition method, the weight controlling unit controls the weight ofthe first neural network unit and the weight of the second neuralnetwork unit according to the step of:

determining whether the predetermined parameter is 1 (S1051); when thepredetermined parameter is 1, reducing the distance between the firstfeature and the second feature in the feature space by the weightcontrolling unit 15 (S1052);

when the predetermined parameter is 0, increasing the distance betweenthe first feature and the second feature in the feature space by theweight controlling unit 15 (S1053).

Namely, the weight controlling unit 15 reduces the distance between thefirst feature and the second feature in the feature space when thepredetermined parameter is 1. The weight controlling unit 15 increasesthe distance between the first feature and the second feature in thefeature space when the predetermined parameter is 0.

For example, when the predetermined parameter is set by the user to be1, the first feature and the second feature should be classified to bethe same gesture event. When the predetermined parameter is set by theuser to be 0, the first feature and the second feature should beclassified to be two different gesture events.

Moreover, the user may further set the predetermined parameter to be 0,and the user may perform two different gestures. The weight controllingunit 15 may increase the distance between the first feature and thesecond feature in the feature space.

With reference to FIG. 4, the gesture recognition method furtherincludes step of:

receiving a sensing signal from the sensor 10 to calculate a sensingfeature by the first neural network unit 11 (S401);

receiving an anchor signal from a database to calculate a referencefeature by the second neural network unit 12 (S402);

generating a distance between the sensing feature and the referencefeature in the feature space by the similarity analysis unit 14 (S403);

determining whether the distance between the sensing feature and thereference feature is smaller than the threshold value (S404);

when the distance between the sensing feature and the reference featureis smaller than the threshold value, classifying a gesture event by thesimilarity analysis unit 14 (S405).

With reference to FIG. 5, the gesture recognition system furtherincludes a database 16. The database 16 stores at least one anchorsignal.

The first neural network unit 11 receives a sensing signal from thesensor 10 to calculate a sensing feature, and the second neural networkunit 12 receives the anchor signal from the database 16 to calculate areference feature.

The similarity analysis unit 14 further generates a distance between thesensing feature and the reference feature in the feature space. When thedistance between the sensing feature and the reference feature issmaller than a threshold value, the similarity analysis 14 classifiesthe gesture event.

For example, when the user wants to use the present invention toclassify the gesture event, the user may perform a gesture, and thesensor 10 may sense the gesture of the user to generate the sensingsignal.

The similarity analysis unit 14 may calculate the distance between thesensing feature from the sensor 10 and the reference feature from thedatabase 16 in the feature space to determine the gesture event.

When the distance between the sensing feature from the sensor 10 and thereference feature is greater than the threshold value, the distancebetween the sensing feature and the reference feature in the featurespace is too long. Therefore, the gesture of the user cannot beclassified.

However, when the distance between the sensing feature from the sensor10 and the reference feature is smaller than the threshold value, thedistance between the sensing feature and the reference feature in thefeature space is short enough. Therefore, the gesture of the user can beclassified, and the similarity analysis unit 14 classifies the gestureevent.

In the above embodiments, the first neural network unit 11 and thesecond neural network unit 12 execute convolutional neural networks(CNNs) or recurrent neural networks (RNNs), the first training signaland the second training signal are Range Doppler Image (RDI) signals,and the distance determined by the similarity analysis unit 14 iscalculated by a contrastive loss function.

Even though numerous characteristics and advantages of the presentinvention have been set forth in the foregoing description, togetherwith details of the structure and function of the invention, thedisclosure is illustrative only. Changes may be made in detail,especially in matters of shape, size, and arrangement of parts withinthe principles of the invention to the full extent indicated by thebroad general meaning of the terms in which the appended claims areexpressed.

What is claimed is:
 1. A gesture recognition method using siamese neuralnetwork, comprising steps of: controlling weight of a first neuralnetwork unit and weight of a second neural network unit to be the sameby a weight sharing unit; receiving a first training signal from asensor to calculate a first feature by the first neural network unit;receiving a second training signal from the sensor to calculate a secondfeature by the second neural network unit; determining a distancebetween the first feature and the second feature in a feature space by asimilarity analysis unit; controlling the weight of the first neuralnetwork unit and the weight of the second neural network unit throughthe weight sharing unit to adjust the distance between the first featureand the second feature in the feature space according to a predeterminedparameter by a weight controlling unit; receiving a sensing signal tocalculate a sensing feature by the first neural network unit; receivingan anchor signal to calculate a reference feature by the second neuralnetwork unit; generating a distance between the sensing feature and thereference feature in the feature space by the similarity analysis unit;when the distance between the sensing feature and the reference featureis smaller than a threshold value, the similarity analysis unitclassifies a gesture event.
 2. The gesture recognition method as claimedin claim 1, wherein the weight controlling unit reduces the distancebetween the first feature and the second feature in the feature spacewhen the predetermined parameter is
 1. 3. The gesture recognition methodas claimed in claim 1, wherein the weight controlling unit increases thedistance between the first feature and the second feature in the featurespace when the predetermined parameter is
 0. 4. The gesture recognitionmethod as claimed in claim 1, wherein the first neural network unit andthe second neural network unit execute convolutional neural networks(CNNs) or recurrent neural networks (RNNs).
 5. The gesture recognitionmethod as claimed in claim 1, wherein the first training signal and thesecond training signal are Range Doppler Image (RDI) signals.
 6. Thegesture recognition method as claimed in claim 1, wherein the distancegenerated by the similarity analysis unit is calculated by a contrastiveloss function.
 7. A gesture recognition system using siamese neuralnetwork, comprising: a sensor, sensing a first training signal and asecond training signal; a first neural network unit, electricallyconnected to the sensor to receive the first training signal, andcalculate a first feature; a second neural network unit, electricallyconnected to the sensor to receive the second training signal, andcalculate a second feature; a weight sharing unit, electricallyconnected to the first neural network unit and the second neural networkunit to control weight of the first neural network unit and weight ofthe second neural network unit to be the same; a similarity analysisunit, electrically connected to the first neural network unit and thesecond neural network unit to receive the first feature and the secondfeature; wherein the similarity analysis unit determines a distancebetween the first feature and the second feature in a feature space; aweight controlling unit, electrically connected to the similarityanalysis unit and the weight sharing unit; wherein the weightcontrolling unit controls the weight of the first neural network unitand the weight of the second neural network unit through the weightsharing unit to adjust the distance between the first feature and thesecond feature in the feature space according to a predeterminedparameter; wherein: the first neural network unit further receives asensing signal to calculate a sensing feature; the second neural networkunit further receives an anchor signal to calculate a reference feature;the similarity analysis unit further generates a distance between thesensing feature and the reference feature in the feature space; when thedistance between the sensing feature and the reference feature issmaller than a threshold value, the similarity analysis unit classifiesa gesture event.
 8. The gesture recognition system as claimed in claim7, wherein the weight controlling unit reduces the distance between thefirst feature and the second feature in the feature space when thepredetermined parameter is
 1. 9. The gesture recognition system asclaimed in claim 7, wherein the weight controlling unit increases thedistance between the first feature and the second feature in the featurespace when the predetermined parameter is
 0. 10. The gesture recognitionsystem as claimed in claim 7, wherein the first neural network unit andthe second neural network unit execute convolutional neural networks(CNNs) or recurrent neural networks (RNNs).
 11. The gesture recognitionsystem as claimed in claim 7, wherein the first training signal and thesecond training signal are Range Doppler Image (RDI) signals.
 12. Thegesture recognition system as claimed in claim 7, wherein the distancedetermined by the similarity analysis unit is calculated by acontrastive loss function.